(19) 



J) 



Europaisches Patentamt 
European Patent Offic 
Office europeen des brevets 



(12) 



(11) EP 0 902 379 A2 

EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

17.03.1999 Bulletin 1999/11 

(21) Application number: 98307334.7 

(22) Date of filing: 10.09.1998 



(51) lntc|6: G06F 17/24, G06F 17/21, 
G06F 17/27 



(84) Designated Contracting States: 


• Price, Morgan N. 


AT BE CH CY DE DK ES Fl FR GB GR IE IT Lt LU 


Palo Alto, California 94306 (US) 


MC NLPTSE 


• Golovchinsky, Gene 


Designated Extension States: 


Palo Alto, California 94306 (US) 


ALLTLVMKROSI 


• Weiser, Mark D. 




Palo Alto, California 94301 (US) 


(30) Priority: 15.09.1997 US 929427 




(74) Representative: Skene James, Robert Edmund 


(71) Applicant: XEROX CORPORATION 


GILL UENNtNGS & EVERY 


Rochester, New York 14644 (US) 


Broadgate House 




7 Eldon Street 


(72) Inventors: 


London EC2M 7LH (GB) 


• Schim, William N. 


Palo AHo, California 94304 (US) 





(54) A method and system for organizing documents based upon annotations in context 



(57) A document organizing system extracts anno- 
tations made to a document along with the context sur- 
rounding each annotation and organizes the annota- 
tions based upon the annotation attributes and/or con- 
text. The annotations are created by grouping marks 
based upon their proximity in time and space. The doc- 



ument is segmented to determine a minimum context 
associated with each annotation. A list of the annota- 
tions sorted by the attributes are then displayed to the 
user. The context provided by the Invention for each an- 
notation allows the user to fully understand the annota- 
tion. 
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Description 

[0001 ] This invention is directed to a document organ- 
izing system. In particular, this invention is directed to a 
method and a system for organizing documents based 
upon the context of annotations made to those docu- 
ments. 

[0002] When people read paper documents, they of- 
ten make annotations to highlight Interesting or contro- 
versial passages and to record their reactions. Common 
annotations include margin notes, vertical bars, stars, 
circles, underlines, highlights, etc. Two advantages of 
annotating directly on the page are its low overhead and 
convenience. One disadvantage is that the recorded tn- 
fonmation is hidden and inaccessible until the reader re- 
turns to the specific page in the specific document. 
[0003] To avoid this problem, some readers use a 
separate reading notebook to record their annotations. 
A reading notebook is useful because it provides a sep- 
arate summary of what the user has read along with any 
commentary. The advantage of a reading notebook is 
that it permits a quick review of the material because it 
generally has less information to browse and search 
than the original document One disadvantage of a read- 
ing notebook, however, is that the reader must recreate 
the context for each note to fully understand the mean- 
ing of each note. 

[0004] Readers also use note cards to organize 
notes. The advantage of a note card system is that the 
cards can be easily reorganized. However, as with a 
reading notebook, unless the reader recreates it, there 
is no context available to permit the user to fully under- 
stand the notes. Additionally, each note must be cate- 
gorized onto the correct card before it can be recorded. 
[0005] Handwritten notes and keywords are used in 
a system known as 'Marquee" to index video. This sys- 
tem is described in "Marquee: A Toot for Real-Time Vid- 
eo Logging", K. Webber et al.. Proceedings of CHI '94 . 
April 1994. pp. 58-64. incorporated herein by reference 
in its entirety. In "Marquee", notes are synchronized to 
a vkJeo string with time zones that are created with hor- 
izontal line gestures. Keywords are kJentified by the user 
by circling the words and notes that the user has select- 
ed as keywords. The keywords are assigned to the time 
zone in which the keyword Is created. Keywords also 
may be assigned directly by the user by typing the key- 
word In manually. Because the keywords are associated 
with time the user can view an index of time zones and 
go directly to the video by selecting a time zone using 
an index of the previously kientified keyword or annota- 
tions. Although "Marquee" uses annotatkxis to index a 
video document, it does not combine the annotations 
with the document in a visual way. "Marquee" Is thus 
anak>gous to notetaking In a separate notebook rather 
than on the document itself. 

[0006] "Dynomile" is a free-form digital "ink" note- 
book. The digital ink notebook is a pen-based computer 
that the user controls by writing with a pen directly on 



the screen of the computer. The computer senses the 
kx:ation and the positions traversed as the pen moves 
across the display and assigns ink marks that corre- 
spond with the positions of the pen. These Ink marks 

s are called digital ink because the ink is described by the 
computer digitally. Dynomite extracts the ink. assigns 
properties to each ink nrBrk and can present a list of the 
ink marks sorted by the assigned properties. This list is 
known as an ink index. This system is described in co- 

10 assigned and co-pending EP Patent Application No. 
983021 27.0 entitled "System for Capturing and Retriev- 
ing Audio Data and Corresponding Handwritten Notes", 
and "Dynomite: A Dynamically Organized Ink and Audio 
Notebook", by L. Witeox et al.. In CHI "97 Conference 

IS Proceedings. ACM Press, 1997, pp. 186-193, incorpo- 
rated herein by reference in their entireties. This ink in- 
dex shows a "type" of the "ink" along with a time stamp 
and provides links to the original notebook pages. Dyn- 
omite's ink index provides "ink" marks linked to the cor- 

20 responding full notebook page. However. Dynomite or- 
ganizes only the ink notes themselves and not the as- 
sociated information. 

[0007] "ComMentor" Is a platfomri for shared annota- 
tions that attaches text-based comments to locations 

2S within web documents. This system is described in 
"Shared Web Annotations as a Platform for Third-Party 
N^lue- Added Information Providers: Architecture, Pro- 
tocols, and Usage Examples", by M. Roscheisen, etal., 
Technk;al Report STAN-CS-TR-97-1582, Stanford Inte- 

30 grated Digital Library Project, Computer Science De- 
partment, Stanford University, November 1994, Updat- 
ed April 1 995, incorporated herein by reference in its en- 
tirety. Annotations are grouped into sets. A user can fitter 
these sets and tour through documents within a set. A 

35 tour window shows a list of annotations, each annotatk>n 
shown with the document title of the annotated docu- 
ment and a number of annotation attributes. Clicking on 
the annotation causes the display to jump to the source 
document at the position of the annotation. ComMentor 

40 uses filtered annotatkxis to produce lists of read docu- 
ments, but does not support paper-like annotations or 
present lists of annotations In context. 
[0008] Classroom 2000 is a system for capturing a 
lecture using recorded audio, prepared visual materials 

^ and handwritten notes made on a display overlay of 
viewgraphs. This system is described in "Classroom 
2000: Enhancing Classroom Interaction and Review", 
by G. Abowd et al.. In Proceedings of CSCW '96. March 
1996, incorporated herein by reference in its entirety. 

50 Searching the text in the viewgraphs retrieves the view- 
graphs along with the overlakj notes. 
[0009] The Freestyle system, which was developed 
at Wang Laboratories, is a mechanism for sketching and 
writing on screen snapshots or on sheets of electronic 

55 paper. Freestyle records cursor nrtovement and audio as 
well as the handwriting. This system is described in 
"Rapid Integrated Design of a Multimedia Communica- 
tion System, and Human-Computer Interface Design", 
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E. Francik, Marianne Rudtsill et al. (editor). Morgan 
Kaufman Publisliers. Inc.. 1996, incorporated herein by 
reference in its entirety. The result is a dynamic multi- 
media message that can be nrtailed to others. Freestyle 
does not provide the ability to organize the handwritten 
annotations. 

[0010] The PENPOINT operating system for pen- 
based computers, recognizes pen gestures for editing 
and albws arbitrary "ink" marks to be placed on top of 
any document using an "acetate layer'. This system is 
described in "The Power of PENPOINT", by R. Can^ el 
al., Addison-Wesley, Inc., 1991, incorporated herein by 
reference in its entirety. Although both Freestyle and 
PENPOINT support free-form document annotation, 
neither provides any way to retrieve documents based 
upon those annotations. 

[0011] In 1 945 Vannevar Bush described a vision of a 
mesh of traits running through a mechanized private file 
and Itorary or memex in "As We May Think", in Atlantic 
Monthly . July 1 945, pp. 101 -108, incorporated herein by 
reference In its entirety. These trails were produced as 
part of the reading activity, and provided a way to create 
and share personal organizations of infomnation. Bush's 
vtsk>ns were seminal in the development of hypermedia 
systems such as Engelbart's NLS and the World Wtde 
Web. However, hypermedia systems have focused on 
sharing, browsing and more explicit authoring of links, 
not on personal organ izatbn and annotation. 
[001 2] Thus, an annotation system for electronic doc- 
uments is needed that combines the advantages of 
marking directly on a document with quick accessibility 
and the flexible organization of marking on note cards 
or in a notebook. 

[0013] This invention provkfes a system and method 
for using digital "ink" for annotatk)ns in context to organ- 
ize a reader's activities. The system and method of this 
invention extracts the contents surrounding and under- 
lying a reader's annotations and presents this informa- 
tion to the reader with links to the full context The an- 
notations in context provided by the system and method 
of this invention pemiits flexible low-overhead organiza- 
tbn of material without adding to the effort of reading 
and notetaking. 

[0014] These and other features and advantages of 
this Inventbn are described in or are apparent from the 
toik>wing detailed descriptk)n of the prefered embodi- 
ments. 

[0015] The preferred embodiments of this invention 
will be described in detail, with reference to the following 
figures, wherein: 

Fig. 1 is a block diagram of the document organizing 
system of this invention; 

Fig. 2 is a flow chart outlining the control routine of 
one embodiment of this invention; 
Fig. 3 shows a document annotated according to 
this invention; 

Fig. 4 shows the annotated portions of the docu- 



ment of Fig. 3 

Fig. 5 shows another view of the annotated docu- 
ment of Fig. 3; and 

Fig. 6 is a f tow chart outlining the annotation control 
s routine of one embodiment ot this invention. 

[0016] Fig. 1 is a block diagram of one embodiment 
of the electronic document organizing system 10 of this 
invention. The system 10 has a processor 12 commu- 
te nk:ating with a display 14. a first storage device 16. a 
second storage device 18 and an input/output interface 
20. The first storage devk:e 1 6 stores a document 22 
displayable on the display 1 4. The input/output interface 
20 communicates with any number of conventional tn- 

15 put/output devices 24 such as a mouse 26, a keyboard 
28 and/or a pen-based device 30. A user manipulates 
the input/output devices 24 to annotate the document 
22 when displayed on the display 14. The system 10 
then stores these annotations 32 in the second storage 

20 device 18, 

[001 7] As shown in Fig. 1 . the system 1 0 is preferably 
implemented using a programmed general purpose 
computer. However, the system 10 can also be imple- 
mented using a special purpose computer, a pro- 

2S grammed microprocessor or microcontroller and any 
necessary peripheral integrated circuit elements, an 
ASIC or other integrated circuit, a hardwired electronic 
or logic circuit such as a discrete element circuit, a pro- 
grammable togic device such as a PLD, PLA. FPGA or 

30 PAL, or the like. In general, any devk;e on which a finite 
state machine capable of implementing the flowchart 
shown in Fig. 2 can be used to implement the system 1 0. 
[0018] Additionally, as shown in Fig. 1 , the memories 
16 and 18 are preferably implemented using static or 

35 dynamic RAM. However, the memories 16 and 18 can 
also be implemented using a floppy disk and disk drive, 
a writable optical disk and disk drive, a hard drive, flash 
memory or the like. Additionally, it shoukj be appreciated 
that the merrxjries 16 and 18 can be either distinct por- 

<o lions of a single menwry or physically distinct memories. 

[001 9] Furthermore, it should be appreciated that the 
link 1 7 connecting the memory 16 and the processor 1 0 
can be a wired or wireless link to a network (not shown). 

45 The network can be a local area networic, a wide area 
network, an intranet, the internet or any other distributed 
processing and storage network. In this case, the elec- 
tronic document 22 is pulled from a physically remote 
memory 16 through the link 17 for processing in the 

so processor 10 according to the method outlined below 
In this case, the electronic document 22 can be stored 
locally in a portion of the memory 1 8 or some other mem- 
ory (not shown) of the system 10. 
[0020] The method of this invention includes three 

S5 distinct processes. First, the reader nr^kes annotatkxis 
on a displayed document, and the annotations are ex- 
tracted along with their context. Second, the system as- 
sociates a nunr^er of attributes with the annotations in 
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order to facilitate retrieval of the annotationG and/or tlie 
underlying annotated documents. Third, the reader 
views collections of the annotations in context, where 
the collections are organized by those attributes. 
[0021] The system 10 records annotations on elec- 
tronic documents. A preferred interface for entering the 
annotations is a pen-based computer, where the reader 
'writes' directly on the electronic document On a desk- 
top computer without a pen. clicking a mouse in a margin 
might create a text overlay box to create the annotation. 
The system 10 may also support a number of different 
styles of marking. For example, these styles can include 
swiping with a highlighter pen. underlining text, vertical 
bars in the margin, circled regbns. and margin notes. 
[0022] Fig. 2 Is a flow chart outlining a control routine 
of one embodiment of the invention. The control routine 
starts at step S100 and proceeds to step S110. where 
the user marks on the display of the document with dig- 
ital ink to annotate it. The control routine then proceeds 
to step S1 20, where the system groups the marks of the 
digital ink by time and/or space into collections of marks, 
treated as a single annotatton as will be described in 
more detail below. Next, the control routine proceeds to 
step S130. where the system determines the minimum 
context for each annotation. The system has a minimum 
context that determines how much of the document that 
surrounds the annotation is to be associated with the 
annotation. The minimum context may be predeter- 
mined as a user preference to be a few words, a sen- 
tence, a paragraph or any other amount in accordance 
with the user's preferences. The minimum context can 
be displayed to the user as a bounding box around the 
minimum context. The bounding box encloses the 
bounding region and the minimum context is defined as 
the content enclosed within the bounding regbn of the 
corresponding annotatbn. Segmentatbn procedures 
are applied to the document to divide it into graphical 
components, e.g., lines of text, sentences, paragraphs 
and figures. Given the minimum context, the control rou- 
tine expands the context to include all of the nearby seg- 
ments. With this procedure, the context may include a 
couple of lines, the surrounding sentence, or the entire 
surrounding paragraph. Fig. 5 shows a bounding box 34 
with the context around a circle annotation 33. 
[0023] The annotatton control routine Is shown In Fig. 
6. The control routine starts at step S200 and proceeds 
to step S210 where the user selects and opens an elec- 
tronic document. The user then starts marking on the 
document at step S220 and creates digital ink. The sys- 
tem then determines at step S230 if the new ink is close 
enough in time and space to be associated with previous 
ink marks. The system has time and space thresholds 
that may be predetermined or adjusted in accordance 
with a user's preferences. If the system determines at 
step S230 that the ink marks are not separate the sys- 
tem proceeds to step S240 where the user continues to 
mark. As each mark is entered by the user steps S230 
and S240 are repeated until the system determines that 



the new ink is separated enough by time and space to 
proceed to step S250. At step S250 the ink marks are 
grouped together as a single annotatbn and at step 
S260 the context for the annotation is determined and 

s the attributes are assigned to the annotatk)n. The con- 
trol routine then proceeds to step S270 where the sys- 
tem determines if a new mark has been input. If a new 
mark has been input the control routine returns to step 
S230. If no new mark is entered at step S280 then the 

10 annotations are organized and displayed. The control 
routine then stops at step S290. 
[0024] For some special annotation formats such as 
those shown in Fig. 5, the control routine determines the 
context slightly differently. For margin bars 36 and other 

15 riotes in the margin 38, the system ignores the horizontal 
distance when finding nearby segments. Thus, all ver- 
tically adjacent material is included in the contexts 40 
and 42, respectively. For the line cal louts and circle call- 
outs, the control routine determines the minimum con- 

20 texts and from the underlined or circled text, etc., ignor- 
ing the ink in the calk)ut gesture. 
[0025] After the context of each annotatbn has been 
determined, the control routine proceeds to step S140, 
where the control routine assigns attributes to the anno- 

25 tattons in at least one of three ways: 1 ) attributes entered 
by the user; 2) attributes inherited from the document's 
attributes; and 3) implicit or explicit attributes derived 
from the annotatfons themselves. 
[0026] The user may enter attributes by interacting 

50 with a dialog box or by selecting from a nrrarking menu, 
or by selecting a special pen. Example, attributes de- 
rived from the annotations themselves include 'agree', 
"disagree*, 'good idea", and "follow-upV In addition, an- 
notation gestures such as "exclamatbn point" and 

35 •questkxi mark" may be interpreted to mean "good idea' 
and 'questionable' by the system as they are entered 
on the page. Attributes may also be entered implicitly, 
the most important of which is the date and time that the 
annotation was made and the page number at the an- 

40 notation. Another implteit attribute Is the form of the an- 
notation, e.g., highlight, circle, marginal note, etc. 
[0027] Attributes may also be inferred from docu- 
ments. In the system 10, the electronic documents are 
already associated with a variety of attributes, such as 

<5 creation date, author, providence and title. 

[0028] After the attributes are assigned to each anno- 
tation at step S140, the control routine proceeds to step 
SI 50, where the annotations are organized, ordered or 
ranked by the assigned attributes. Subsequently, the 

50 control routine proceeds to step S1 60, where the anno- 
tations are displayed for the user. The control routine 
then proceeds to step SI 70, where the control routine 
stops. 

[0029] The system 1 0 visually presents the annota- 
55 tions in context using different list views. Lists are or- 
dered or filtered by the attributes described above. The 
system 10 allows the reader to navigate between these 
views and the underlying electronic documents. Exam- 
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pies of ordered lists include: 

1) Ordered by time. This view is analogous to a 
reader's notebook, but also automatically Includes 
the context of each annotation, as shown In Fig. 4, 
without further effort by the user. 

2) Filtered by attributes. Passages across a number 
of documents are listed in one view; 

3) Filtered by the type of adjacent material. For ex- 
ample, annotations of pictures along with the pic- 
tures themselves; and 

4) Filtered by the content of adjacent material. For 
example, annotated passages mentioning patent 
leather shoes are ranked in relatedness using 
known Information retrieval techniques. 

[0030] It is to be understood that the term annotation 
as used herein is intended to include text, digital ink, 
audio, video or any other Input associated with a docu- 
ment It is also to be understood that the term document 
is intended to include text, video, audio and any other 
media and any combination of media. Further, it Is to be 
understood that the term text is intended to Include text, 
digital ink. audio, video or any other content of a docu- 
ment to include the document* s structure. 



Claims 

1 . A method for displaying context embedded annota- 
tk^ns from at least one document, comprising: 

extracting at least one annotation from the at 
least one document; 

extracting a context portion for each at least 
one annotation from a corresponding one of the 
at least one document; and 
assigning at least one attribute to each at least 
one extracted annotation. 

2. The method of claim 1 . wherein the step of assign- 
ing at least one attribute comprises assigning at 
least one user-defined attribute to each at least one 
annotatbn. 

3. The method of claim 1 , wherein the step of assign- 
ing at least one attribute comprises assigning at 
least one document-based attribute to each at least 
one annotation. 

4. The method of any one of claims 1 to 3, wherein the 
step of extracting at least one annotation compris- 



grouping marks by time and space into at least 
one collection, wherein each collection forms 
an annotation: 

segmenting the at least one document into a 



plurality of segments based on the at least one 
annotation; 

determining a minimum context portion for 
each annotatksn, from the segments; and 
determining the context portion based on the 
segments surrounding the minimum context 

5. The method of any one of claims 1 to 4, further com- 
prising: 

ordering the at least one anriotatton based on 
the at least one attribute; and 
displaying an ordered list of the at least one an- 
notatton along with the corresponding context 

6. An apparatus for displaying annotations from at 
least one document, the apparatus comprising: 

a memory that stores the at least one docu- 
ment; 

a processor that extracts at least one annota- 
tion and a context portion of the at least one 
document corresponding to each at least one 
annotation, that assigns at least one attribute 
to each at least one extracted annotation that 
orders the at least one annotations based on 
the at least one assigned attribute; and 
a display that displays an ordered list of the at 
least one annotations and the at least one cor- 
responding context portion. 

7. The apparatus of claim 6. wherein, for each at least 
one annotation, the processor assigns the at least 
one attribute to that annotatcn based on at least 
one document-based attribute. 

8. The apparatus of claim 6. wherein, for each at least 
one annotation, the processor assigns the at (east 
one attribute to that annotatbn based on at least 
one attribute derived from the context portion cor- 
responding to that annotation. 

9. The apparatus of claim 6, wherein when one of the 
at least one annotation is a mark annotation the 
processor assigns the at least one attribute based 
upon that mark annotatbn. 

1 0. The apparatus of any one of claims 6 to 9. wherein: 



so the processor extracts the at least one annota- 

tion by grouping at least one user generated 
mark on the at least one document based on 
the time of the at least one mark and a location 
within the document of the at least one mark 

ss into at least one collection, wherein each col- 

lection forms a single annotation, the processor 
determines a minimum context portion of the 
corresponding document for each annotatbn; 
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determines the context for each annotation 
based on the annotation and the minimum con- 
text 
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shillings worth of goods; so that it often rttoms into the 
bands of a dealer, before the constuner has spcit the forti- 
etfa part of the money. Where bank notes are issoeci ^or 50^ 3o 
small sums as tweat7shillmgs» as in ScotJaad» paper money 

.37 

notes, it fiB ed a still greater p OT of that drcnlatioiL In the 
cmreociei gl^Worttt An aerica^^ per ynz commonly issued for 
so small aSBrff^i sUUhngJanii almost the whole of 

36 

was issued even for so small s sum as a sixpence* j 

Where the issuing of bank notes for such very small sums | / 
is allowed and commonly practised, many mean people are | • 
both enal#led and encouraged to become bankers. A person 

as a sixpence. But the frequent bankruptcies to which such j 
beggsrly bankers must be liable, may occasion a very con^\/^S 
tidcraUe inoonvenicncy, and sometimes eren a very great z^^;;^^ 
calamity, to many poor peopk who had recdred their notes 
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are ctrcolated tmder tea potmds vaiue^^^in 
tBoney confimf itself verr mnch to the ci 
the ckalera. jWbeg a tm pound bank note oonea ino the 
hands of Tconsnniert he is (cncxally obliged to chance it 
at the first shop where he has occasion to tmrdase fire 
shillings worth ol goods;! so that it often wtnrBs into the ^^jt^' 
hands of a deato; oefoie the constaner ha * «r*^ **** ,1"'*^TT 
eth part of the mooSyJ Where bank notes are issoed lor so 
"SSB^Si as twenty shillings^ as in Scotland, paper money 
extends itself to a contidersble part of the drcalation be- 
tween dealers and coosnmen^ Before die act of parliament; 
wUeh put a stop to the ctmlation of ten and five sMQiac, 

MWii filkli,itiB row j«t 9f tt« mrafatigai I ig ^ 

<orth Aa ericauft a per was cc rmrnnTT 
^ ^11 ^^uuk m A jlililiiiT AlU aifnfttt the whole of 
,that drcolatioo. I In some paper cnirendea of Yorkshire, it 
was isined eren for f | mn as a sixp ence. 



Where the issoi^ of bank noto^or^ rach very small sums 
is allowed and comrnnnfy practised, many mean people are 
both enabled and cacooTagyd to become bankea. | A person^ 
whose promissory note for five pounds, or eren for twenty 
shfllings, wotild be rejected by eveiy body, will get it to be 
received wtAflutjfff^plg when it h for ^ ^ 

as a aixoettce. I But the frequent bankruptcies to which such 
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beggarly banketa must be liable, may occasion a very coir-l 
sidcrable ineomreniency, and sometimes even a very greatj 
calanutyt to manr poor people who had receivtd thdr tiotesl 

in paymentLj ^ K-'i 

tt were better, perhaps, that no bank notes wer e issu ed 
la any part of the kindom for a smaller sum thsn five poonda. 
Pspcr money wouU then, probably, confine itself, in every 
pert of the kindom, to the circulatioa between the different 
dealers, as much as it does at present in L on d on , where no 
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